4 research outputs found
Deep Vision for Prosthetic Grasp
Ph. D. ThesisThe loss of the hand can limit the natural ability of individuals in grasping and
manipulating objects and affect their quality of life. Prosthetic hands can aid the
users in overcoming these limitations and regaining their ability. Despite considerable
technical advances, the control of commercial hand prostheses is still limited
to few degrees of freedom. Furthermore, switching a prosthetic hand into a desired
grip mode can be tiring. Therefore, the performance of hand prostheses should
improve greatly.
The main aim of this thesis is to improve the functionality, performance and flexibility
of current hand prostheses by augmentation of current commercial hand prosthetics
with a vision module.
By offering the prosthesis the capacity to see objects, appropriate grip modes can
be determined autonomously and quickly. Several deep learning-based approaches
were designed in this thesis to realise such a vision-reinforced prosthetic system.
Importantly, the user, interacting with this learning structure, may act as a supervisor
to accept or correct the suggested grasp. Amputee participants evaluated the
designed system and provided feedback.
The following objectives for prosthetic hands were met:
1. Chapter 3: Design, implementation and real-time testing of a semi-autonomous
vision-reinforced prosthetic control structure, empowered with a baseline convolutional
neural network deep learning structure.
2. Chapter 4: Development of advanced deep learning structure to simultaneously
detect and estimate grasp maps for unknown objects, in presence of
ambiguity.
3. Chapter 5: Design and development of several deep learning set-ups for concurrent
depth and grasp map as well as human grasp type prediction.
Publicly available datasets, consisting of common graspable objects, namely Amsterdam
library of object images (ALOI) and Cornell grasp library were used within
this thesis. Moreover, to have access to real data, a small dataset of household
objects was gathered for the experiments, that is Newcastle Grasp Library.EPSRC, School of Engineering Newcastle University
Deep learning-based artificial vision for grasp classification in myoelectric hands
Objective. Computer vision-based assistive technology solutions can revolutionise the quality of care for people with sensorimotor disorders. The goal of this work was to enable trans-radial amputees to use a simple, yet efficient, computer vision system to grasp and move common household objects with a two-channel myoelectric prosthetic hand. Approach. We developed a deep learning-based artificial vision system to augment the grasp functionality of a commercial prosthesis. Our main conceptual novelty is that we classify objects with regards to the grasp pattern without explicitly identifying them or measuring their dimensions. A convolutional neural network (CNN) structure was trained with images of over 500 graspable objects. For each object, 72 images, at intervals, were available. Objects were categorised into four grasp classes, namely: pinch, tripod, palmar wrist neutral and palmar wrist pronated. The CNN setting was first tuned and tested offline and then in realtime with objects or object views that were not included in the training set. Main results. The classification accuracy in the offline tests reached for the seen and for the novel objects; reflecting the generalisability of grasp classification. We then implemented the proposed framework in realtime on a standard laptop computer and achieved an overall score of in classifying a set of novel as well as seen but randomly-rotated objects. Finally, the system was tested with two trans-radial amputee volunteers controlling an i-limb UltraTM prosthetic hand and a motion controlTM prosthetic wrist; augmented with a webcam. After training, subjects successfully picked up and moved the target objects with an overall success of up to . In addition, we show that with training, subjects' performance improved in terms of time required to accomplish a block of 24 trials despite a decreasing level of visual feedback. Significance. The proposed design constitutes a substantial conceptual improvement for the control of multi-functional prosthetic hands. We show for the first time that deep-learning based computer vision systems can enhance the grip functionality of myoelectric hands considerably
Dynamic Scene Graph Representation for Surgical Video
Surgical videos captured from microscopic or endoscopic imaging devices are
rich but complex sources of information, depicting different tools and
anatomical structures utilized during an extended amount of time. Despite
containing crucial workflow information and being commonly recorded in many
procedures, usage of surgical videos for automated surgical workflow
understanding is still limited.
In this work, we exploit scene graphs as a more holistic, semantically
meaningful and human-readable way to represent surgical videos while encoding
all anatomical structures, tools, and their interactions. To properly evaluate
the impact of our solutions, we create a scene graph dataset from semantic
segmentations from the CaDIS and CATARACTS datasets. We demonstrate that scene
graphs can be leveraged through the use of graph convolutional networks (GCNs)
to tackle surgical downstream tasks such as surgical workflow recognition with
competitive performance. Moreover, we demonstrate the benefits of surgical
scene graphs regarding the explainability and robustness of model decisions,
which are crucial in the clinical setting